# Visual Language Models

Ollama-OCR is an optical character recognition (OCR) model based on Ollama that can extract text from images. It leverages advanced visual language models such as LLaVA, Llama 3.2 Vision, and MiniCPM-V 2.6 to provide high-accuracy text recognition. This model is highly useful for scenarios requiring text information extraction from images, such as document scanning and image content analysis. It is open-source, free to use, and easily integrable into various projects.

vision-parse is a tool that uses visual language models (Vision LLMs) to convert PDF documents into well-formatted Markdown content. It supports multiple models including OpenAI, Llama, and Gemini, intelligently recognizing and extracting text and tables while preserving the document's hierarchy, style, and indentation. The main advantages of this tool include high-precision content extraction, format retention, multi-model support, and local model hosting, making it suitable for users requiring efficient document processing.

POINTS-Qwen-2-5-7B-Chat integrates the latest advancements and techniques in visual language models, proposed by researchers at WeChat AI. It significantly enhances model performance through techniques like pre-training dataset filtering and model ensembling. This model has shown outstanding performance in multiple benchmark tests, representing a significant advancement in the field of visual language models.

DeepSeek-VL2 is a series of large Mixture-of-Experts visual language models, showing significant improvements over its predecessor, DeepSeek-VL. This series exhibits exceptional performance in tasks such as visual question answering, optical character recognition, document/table/chart understanding, and visual localization. DeepSeek-VL2 includes three variants: DeepSeek-VL2-Tiny, DeepSeek-VL2-Small, and DeepSeek-VL2, with 1.0B, 2.8B, and 4.5B active parameters, respectively. Compared to existing open-source dense and MoE base models with similar or fewer active parameters, DeepSeek-VL2 achieves competitive or state-of-the-art performance.

Florence-VL is a visual language model that enhances the processing capabilities of visual and language information by introducing generative visual encoders and deep breadth fusion technology. The significance of this technology lies in its ability to improve machines' understanding of images and text, achieving better performance in multimodal tasks. Florence-VL is developed based on the LLaVA project, providing code for pre-training and fine-tuning, model checkpoints, and demonstrations.

PromptFix is a comprehensive framework that enables diffusion models to perform various image processing tasks by following human instructions. This framework constructs a large-scale instruction-following dataset, proposes high-frequency guided sampling methods to control the denoising process while retaining high-frequency details in unprocessed areas, and designs auxiliary prompt adapters that enhance text prompts using visual language models, increasing the model's task generalization capabilities. PromptFix outperforms previous methods across various image processing tasks and demonstrates superior zero-shot capabilities in blind restoration and compositional tasks.

ColPali is an efficient document retrieval tool based on visual language models, simplifying the retrieval process by directly embedding images of document pages. Leveraging the latest visual language model technology, particularly the PaliGemma model, ColPali improves retrieval performance through late interaction mechanisms for multi-vector retrieval. This technology not only accelerates indexing speed and reduces query latency, but also excels in retrieving documents containing visual elements such as charts, tables, and images. ColPali introduces a new paradigm of 'visual space retrieval' in the field of document retrieval, enhancing the efficiency and accuracy of information retrieval.

DriveVLM is an autonomous driving system that leverages visual language models (VLMs) to augment scene understanding and planning capabilities. The system employs a unique combination of reasoning modules, encompassing scene description, scene analysis, and hierarchical planning, to enhance comprehension of complex and long-tail scenarios. Addressing the limitations of VLMs in spatial reasoning and computational demands, DriveVLM-Dual was developed as a hybrid system, integrating the strengths of DriveVLM with traditional autonomous driving pipelines. Experiments on the nuScenes and SUP-AD datasets demonstrate the effectiveness of DriveVLM and DriveVLM-Dual in handling complex and unpredictable driving conditions. Ultimately, DriveVLM-Dual has been deployed in production vehicles, validating its efficacy in real-world autonomous driving environments.

MMStar is a benchmark dataset designed to assess the multimodal capabilities of large visual language models. It comprises 1500 carefully selected visual language samples, covering 6 core abilities and 18 sub-dimensions. Each sample has undergone human review, ensuring visual dependency, minimizing data leakage, and requiring advanced multimodal capabilities for resolution. In addition to traditional accuracy metrics, MMStar proposes two new metrics to measure data leakage and the practical performance gains of multimodal training. Researchers can use MMStar to evaluate the multimodal capabilities of visual language models across multiple tasks and leverage the new metrics to discover potential issues within models.